TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect

AbstractPre-trained models have accomplished high performances with the introduction of Transformers like Bidirectional Encoder Representations from known for BERT. Nevertheless, most these proposed been trained on represented languages (English, French, German, etc.) and few target under-represented dialects.This work introduces a feasibility study pre-training language based Tunisian dialect as an languages. The model is evaluated identification task, sentiment analysis reading comprehension question-answering task. Results demonstrate that, instead using datasets traditional sources (Wikipedia, articles, etc.), noisy web crawled data more convenient such dialect. Additionally, experiments show that reasonably small-scale dataset conducts to similar or better achievements when large-scale TunBERT reach enhance state art in all three downstream tasks. pre-trained named used fine-tuning step are publicly released.KeywordsTransformersLanguage modelsUnder-represented languagesTunBERTBERT